Non - Linear Algorithms for Parametric Markov Programming

General idea of nonlinear correcting factors that was successfully applied to accelerate and derive algorithms for many linear and nonlinear problems is used to develop several efficient procedures for solving linear systems of equations based on the Jacobi algorithm. Results of the theoretical study and extensive computer experiments for stochastic matrices are presented and analyzed to determine the conditions under which the studied algorithms converge, and the areas of their maximum convergence rate are identified. Applications of the algorithms to Markov decision processes are discussed. Many problems in computer science and operations research can be formulated as Markov Decision Processes (MDP) [1]. Examples include routing, optimal stopping, target, replacement, maintenance and repair, and inventory problems as well as optimal control of queues and stochastic scheduling. The utility function of total expected discounted rewards is commonly used in MDPs with finite and action spaces [1]. In this case, the optimality equation takes the following form: ( ) ( ) max V R S V α α ω α = ⎡ + ⎤ ⎣ ⎦ , (1) where V is a value (state) vector, ( ) R α is a reward vector, ω is a scalar discount factor, ( ) S α is a transition probability matrix, and α is a policy (control vector). The objective is to find the optimal policy α and corresponding value vector V, which represents the maximum expected discounted sum of future rewards. The main approaches to solving problem (1) are policy iteration, value iteration, and linear programming [1]. Both policy iteration and value iteration methods strongly depend on the efficiency of algorithms to solve a linear system of equations (LSE). Policy iteration algorithm solves equation ( ) ( ) ( ) ( ) i i V R S V α ω α = + , (2) for every iteration i of policy α . Value iteration algorithm calculates estimates for next value vector iterates using equation ( ) ( ) ( ) ( ) 1 max i i V R S V α α ω α + ⎡ ⎤ = + ⎣ ⎦ . (3) The main approaches to accelerate the convergence of algorithm (3) are similar to those for traditional LSE: Gauss-Seidel, Successive Overrelaxation (SOR), etc. [1]. As a result, faster LSE algorithms can help accelerate existing optimization algorithms and broaden the application area of MDPs. Mathematically a linear system of equations can be formulated as follows: x = Ax + b, (4) where A is a known coefficient matrix, b is a known vector, and x is an unknown solution vector. Two major directions to solving LSE are commonly recognized: direct and iterative [2]. Direct methods [3] often involve factorization, such as Gaussian elimination, and forward and backward substitution on the vector b.